…docling-project#157)

`images=embedded` did nothing for scanned input: the browser pipeline built
its page with `PdfPage::from_cells`, which carries no bitmap, so every
picture region came out as `<!-- image -->` with no pixels to inline — the
option looked broken to anyone converting a scan.

The native pipeline already crops each `picture` region out of the rendered
page; only the feature gate stopped the browser from doing the same. Since
`ocr-prep` is a subset of `ml`, moving the page-bitmap field, its import and
`crop_region` from `ml` to `ocr-prep` gives the wasm build the identical code
path and leaves both the native (`ml`) and text-layer-only (no `ocr-prep`,
no field, no crop) builds exactly as they were.

The browser hands its rasterized page over through the new
`PdfPage::from_cells_with_image`, and the `images` choice is threaded through
`ScannedConverter::finish` / `convert_scanned_image` and the JS layers, so
both conversion paths now offer the same output grammars *and* the same
picture handling.

Verified: a new unit test builds a page from a host-supplied bitmap and
asserts the assembled picture carries a PNG cropped to region × scale
(100x60); the native docling/docling-pdf/docling-core suites (117 tests)
stay green, and the declarative path re-checked in headless Chromium
(placeholder vs embedded, md/json/doclang, preview, text-layer PDF).

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

…erting

Two reports, one cause. The preview checkbox was the only control left live
while a conversion ran, and its handler was wired straight to run(). Clicking
it mid-conversion therefore started a second run on top of the first; the
first then reached finishDoc, which clears the pipeline's single in-flight
document, and the second dereferenced it — surfacing as "Cannot read
properties of null (reading 'tf')".

- The busy/ready freeze now covers every control (a named CONTROLS list, so
  a future addition can't be forgotten again), preview included.
- run() takes a re-entrancy guard, so the UI is no longer the only thing
  standing between us and two concurrent documents.
- Toggling preview only re-renders the output we already have. It never
  reconverted meaningfully before either, but on a scanned document doing so
  costs another ~8 s per page — now it is free.
- addPage/finishDoc reject with a real message when no document is open,
  instead of a null dereference.

Verified in headless Chromium during an actual long-running conversion: all
seven controls read disabled, clicking preview while frozen leaves exactly
one run in flight, toggling preview afterwards leaves the timing line
untouched (no reconversion) and renders the embedded images. The browser also
caught a TDZ slip in the first cut of this change (CONTROLS used above its
declaration), fixed here.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

…docling-project#157)

**Selective TableFormer.** The encoder runs once per table region and costs
seconds, but many tables need no model at all. Each table is now first
reconstructed geometrically — free, from the OCR cell positions — and
TableFormer is invoked only when that grid looks unreliable.

The test comes from how `reconstruct_table` derives columns: it clusters cell
*left edges*, exact on a clean grid but prone to splitting one real column
into several when entries are not left-aligned, leaving a wide mostly-empty
table (the "spurious empty columns" a scanned invoice produced). So
`geometric_table_is_reliable` trusts a grid only when it is dense (>=60% of
cells carry text) and has no column that a single row uses; everything else,
including degenerate shapes, goes to the model. Deliberately one-sided: it
only ever skips work on grids that are plainly well-formed.

This lives in the browser path alone — the native pipeline runs TableFormer
on every region as before, so its conformance output is untouched. The
predicate and the reconstruction are shared code (`assemble` is now public
under `ocr-prep`) and covered by a unit test over both shapes: a genuine
4-column grid is trusted, the split-column invoice shape is not.

**GitHub Pages.** A workflow builds the wasm and publishes `www/` on every
push to master, so the demo is reachable from any device with no checkout;
the root README and the crate README link to it. No models are published —
the page resolves them at runtime (device file, same-origin ./models/, or
Hugging Face), so declarative conversion and text-layer PDFs work on arrival
and OCR works as soon as models are supplied. Pages cannot send COOP/COEP,
but www/coi.js installs the service worker that does, so ORT still gets
threads. Enabling it is one repo setting: Settings -> Pages -> Source:
"GitHub Actions".

Verified: docling/docling-pdf/docling-core/docling-wasm suites green (152
tests) so the shared-code change moved nothing native; pdf-text-only and
no-default feature builds still compile; the page re-checked end to end in
headless Chromium.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

Investigating a cell-merge report from the browser pipeline ("Telefon" and
"0676/2000" landing in one TableFormer cell) meant knowing how coarse the
OCR-derived word boxes actually are, since those are the one input the browser
path does not share with the native one (no pdfium text layer — words come
from segment_words' ink projection).

Measured: the split happens at a gap of 0.57-0.60 x line height, consistently
across line sizes. That is the floor on how tight a table's columns may be
before two entries become a single box — and a single box can only ever match
one table cell. Pin it, so the constant is not silently retuned.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

Telling "this PDF genuinely needs OCR" apart from "our pure-Rust parser
missed a text layer pdfium can read" had no offline reproduction: the CLI's
--no-ocr goes through pdfium's extraction, while a wasm build calls
convert_text_layer (lopdf content-stream parsing). They are different
engines, so --no-ocr succeeding says nothing about what the browser will do.

This example calls convert_text_layer directly and reports whether the result
is empty — which is exactly the condition that makes the browser demo fall
back to OCR. Built with --no-default-features it pulls neither pdfium nor
onnxruntime.

Checked both ways: a corpus text-layer PDF reports 24 nodes ("the browser
would use this directly"), a corpus scan reports 0 ("a browser build would
fall back to OCR here").

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

… text

Confirmed on a real invoice that convert_text_layer returns nothing where
pdfium reads the layer fine, so the browser pays 10 s of OCR for a document
it could convert in milliseconds. textparse cannot say why: every failure
path is `let Ok(doc) = Document::load_mem(..) else { return empty }`, so an
encrypted or damaged PDF is indistinguishable from a scan.

Report the three stages separately — lopdf load (with page count, version and
encryption), raw text lines out of the content streams, then the assembled
document. That splits the three explanations that all look identical today:
the file would not load, the text is there but undecodable (font encoding, an
unhandled operator or filter), or the page really is a scan.

Checked both ends: a corpus text-layer PDF reports load ok / 128 lines / 24
nodes, a corpus scan reports load ok / 0 lines / 0 nodes — the shapes that
distinguish "cannot read it" from "nothing to read".

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

A real invoice fails at the very first stage — lopdf reports "invalid file
trailer" while pdfium reads the same file fine — so the browser spends ~10 s
on OCR for a document whose text layer is right there. The failure says
nothing about which part of the cross-reference machinery is wrong, and the
file is personal, so the diagnosis has to come from the reporter's own
machine.

On a load failure the example now dumps what the parser tripped over: which
markers exist (trailer / startxref / xref / /XRef / %%EOF), how many bytes
trail the final %%EOF, the last startxref block, and the bytes at the offset
it points to. A classic xref table, an xref stream, an offset past the end of
the file and plain garbage each read differently there and call for different
repairs.

Exercised on two deliberately broken copies of a corpus PDF: appended junk
after %%EOF still loads (lopdf tolerates it), while a corrupted startxref
offset reports "startxref points to 999999999, past the end of the file".

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

A real invoice converted to nothing in the browser while pdfium read it
fine. Traced it to the cross-reference table: the spec fixes an entry at 20
bytes — `nnnnnnnnnn ggggg n` plus a *two*-byte EOL — and this generator emits
a bare LF, making each entry 19. lopdf rejects the whole file
("invalid file trailer"), so a perfectly readable text layer was
indistinguishable from a scan and cost ~10 s of OCR for nothing.

Reproduced from first principles: two otherwise identical synthetic PDFs, one
with the spec's 20-byte entries and one with 19, give "loads fine" and the
reporter's exact error message.

textparse now pads such entries back to 20 bytes and reloads. Two things keep
this safe. It only fires where padding cannot move anything the table points
at — a single xref section starting after the last object — so an incremental
update is declined rather than silently shifted. And the repair must prove
itself: the padded bytes are used only if they load, so a mis-repair degrades
to the previous behaviour instead of to garbage. All five load sites in
textparse share the new loader.

Tests build both variants in-process (no binary fixture): the repaired parse
must equal the well-formed one cell for cell, and the repair must decline a
file whose xref precedes an object. The 19-byte case also asserts lopdf still
rejects it unaided, so the workaround gets dropped when upstream catches up.

docling, docling-pdf and docling-core suites stay green (152 tests); the
wasm32 build still compiles.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

The repair either fires or it does not, and a decline looks exactly like a
scan from the outside — so a report of "still going to OCR" cannot be told
apart from "the wasm build is stale". Carry a reason out of
pad_short_xref_entries and surface it through xref_repair_status, which the
text_layer example now prints: repaired, declined (and why: an xref stream,
several xref sections, an object after the table, a malformed entry), or
padded-but-still-unloadable.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

The invoice that started this carries a second, independent defect. Its
content streams declare `/Length 2826` where only 2825 bytes reach
`endstream`, so lopdf reads past the data, does not find the keyword it
expects, and drops the stream — the object comes back as a bare dictionary.
The page then has no content at all, which is why the xref repair alone still
produced zero text: the document loaded, found both pages and all four fonts,
and had nothing to read. pdfium tolerates it by trusting `endstream`.

fix_stream_lengths does the same, and does it without moving a byte: the
corrected number is written over the old digits and padded with spaces, so
every offset in the file — and therefore the cross-reference table — stays
valid. It only ever shrinks a length, since a longer number would not fit.

Loading now tries progressively more repair and accepts a candidate only once
the pages actually carry content — "it loaded" is not evidence, precisely
because a stripped-stream document loads fine. That also fixes an ordering
bug in the previous cut, which returned the first candidate that merely
loaded and so never reached the second repair. A well-formed file returns on
the first attempt and pays for none of this.

On the reporting file: 0 nodes -> 78 nodes, 1908 chars, with the umlauts the
OCR fallback was mangling ("Finanzübersicht", "spätestens").

The regression test builds the defect in-process rather than committing
anyone's invoice, and asserts lopdf still drops the stream unaided, so the
workaround can be retired when upstream handles it.

docling and docling-pdf suites stay green (152 tests); wasm32 still compiles.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

)

Fixing the text-layer parser made this invoice convert in 63 ms instead of
ten seconds of OCR — and lost its tables. That was not a regression but the
shape of the choice the browser offered: the pure text-layer path is exact
and instant but emits flat paragraphs, because headings, tables and pictures
are all things the *layout* model finds; the OCR path has structure but
re-reads text the file already contains, slowly and with recognition errors
("Finanzubersicht", "magenta.at/fag").

DigitalConverter is the third path, and it is what the native pipeline
already does: take the text from the file, run only the layout model over the
rendered pages, then assemble. Region refinement now has the real text cells
to work with (orphan-text recovery and the false-picture drop do nothing on
an empty slice, which is all the OCR path can pass at that point), and
TableFormer stays selective. No recognition model is fetched at all — boot
takes a layoutOnly flag, so a visitor converting digital PDFs never downloads
it.

The page picks the path: a "PDF structure" toggle routes PDFs here, falls
back to OCR only on "no text layer", and leaves everything else alone.

Verified in headless Chromium against the reporting invoice, with ORT and
pdf.js vendored locally since this sandbox cannot reach the CDN: 0 -> 18
table rows, umlauts and the "faq" URL correct, headings and picture
placeholders present — the same document the native pipeline produces.
Timings here (11-17 s cold, layout model load included) say little: the
container is shared and has no GPU.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

…ocling-project#157)

Both structured-path screenshots of the reporting invoice print the
Einspruch paragraph twice. Localized it by elimination: the flat text-layer
output contains the text once, the native pipeline emits it once, and a
native replica of the digital path (same textparse cells, same layout model,
same refinement — pdfium raster instead of pdf.js) also emits it once. So the
pdf.js raster teases overlapping detections out of the model — a block plus
its own halves — and since assembly assigns each region the cells >50% inside
it without consuming them, every such overlap duplicates text.

drop_duplicate_text_claims removes the re-readers: a text-like region is
dropped only when every cell it claims is also claimed by one single other
region — a strict part-of relation, so disjoint neighbours and non-text
labels (a table legitimately shares cells with its caption) are untouched,
and identical claims keep the higher score. The surviving block matches what
the native raster produces on the same page (one merged paragraph). Wired
into the digital browser path only, where the artifact was observed; the
shared assemble stays byte-identical for the native pipeline.

Covered by unit tests over the three shapes: block-plus-halves drops the
halves, disjoint/non-text stays, identical claims dedupe by score. Full
docling-pdf suite green (39 tests), wasm clippy clean.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

The reporting invoice's flat text-layer output opened with `" ==` — its
T-Mobile mark is drawn with a `TeleLogo` Type1 font, stacking the glyphs
encoded as `"` and `==` on top of one another. Nothing in the font metadata
gives that away: the file's *text* fonts carry the same symbolic flag, and
the logo font names its glyphs `quotedbl` &c. (checked by decrypting the
embedded Type1s). The geometry does: two cells with different text where one
lies inside the other on the same line is impossible for prose — so both are
paint, not text.

drop_overpainted_cells applies exactly that containment rule (adjacent words
touch but never contain each other; a same-text double-draw is left alone)
and runs on pdf_text_pages only — the flat/browser path, where there is no
layout model to sink the mark into a `picture` region. The ML pipeline's
byte-pinned text layer (pdf_all_cells) is untouched, and the full corpus
stays green: docling 117, docling-pdf 41, docling-wasm 7.

Refs docling-project#157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT